Repository navigation
Check EESSI-extend install paths, explain node-local refusals, scope build secrets - #65
Merged
Merged
Conversation
EESSI-extend chooses its install tree from EESSI_USER_INSTALL, EESSI_PROJECT_INSTALL, EESSI_SITE_INSTALL and EESSI_CVMFS_INSTALL as it loads. Loaded under Lmod 8.6.19, the module EESSI's software-layer-scripts generate installs below the directory a user or project variable names, into the site tree for the site variable, into EESSI's own tree for the cvmfs one, and under $HOME/eessi with none set. With a user variable left from an earlier session beside a project one, it makes a private user install, umask 077, in the user tree; with the site or the cvmfs variable and another, it refuses to load. A load hook that sets an install variable is too late for its own load and moves the next one. Nothing checked where an install then went. This changes behaviour: - Once EESSI-extend has loaded, a build stops with exit 2, before eb runs, unless EASYBUILD_INSTALLPATH is below <prefix>/versions, and whenever EASYBUILD_INSTALLPATH_SOFTWARE or _MODULES is set. A site hook that set an install path, or an EESSI_CVMFS_REPO with a character such as "-", which EESSI-extend matches as a Lua pattern, sent the install and its logs outside the prefix eb-local checked, lists and cleans of secrets, while eb-local said it installed there. - eb-local eb stops the same way when the install path is outside the tree the install variable selects, when that tree is not known because the EESSI module left EESSI_SITE_SOFTWARE_PATH or EESSI_SOFTWARE_PATH empty, and when EASYBUILD_INSTALLPATH_SOFTWARE or _MODULES is set in the shell or by the load. It passed them on to EasyBuild. - Both warn when loading EESSI-extend sets or unsets an install variable. - A build warns about each install variable it clears from the caller's shell, in the words eb-local doctor uses. - The refusal of more than one install variable names each, with its value, and what EESSI-extend would do with them. It still exits 3. The Bash 1.x script is frozen and keeps its behaviour, so the new messages are tested against the Zig build only. The test doubles now set the install path EESSI-extend sets. The README says what the checks do and links to a new explanation page, written in Org under docs/orgmode and exported to the MyST page Sphinx builds by docs/export.el, which needs only the Markdown backend Org ships. tests/test_docs.py checks that the two agree where Emacs is installed.
A build, eb-local clean and eb-local doctor refused a node-local mount or an install prefix with messages such as "allocation node-local root must be an owned writable directory." or "installation, source and log paths must use durable owned storage.", which named neither the path, nor the setting, nor the reason. Each refusal now has one form. The first line says what was refused and why, then what the command did not do: "nothing was built", "nothing was removed", or "a build would stop here" for --explain. The next line, indented, says what to change: the setting, with its value and where it was set as eb-local config shows it, or the variable, option or command. eb-local doctor puts the same on its FAIL line, with "builds refuse it", as its line for a refused prefix already had. storage.jobRoot takes the node-local settings with where each was set, and returns what to change beside the message. The checks, their order and what they accept are unchanged; a symbolic link, a missing mount and one that is not a directory, refused before with one message, now each have their own. Changed behaviour, as this item needs: - the text of these refusals on stderr and of the doctor FAIL lines; exit statuses are unchanged; - a 1.x eb-local.conf is converted to TOML line for line, so that eb-local config and the errors give each setting its own line, not one in the converted text; - a workspace that cannot be made names the error the kernel gave, where makeWorkspace returned AccessDenied for every failure, and the umask is no longer left cleared when its path is too long. docs/reference/refusals.md, written from Org, lists each refusal. The README's paragraph on node-local storage moves to "Where things go" and links to it.
Every build read every secret the configuration defines and put each in eb's environment, so a build that used none of them still carried them all, and one secret whose source could not be read stopped every build. build.secrets, a list of secret names under [build] in any configuration file or profile, and EB_LOCAL_BUILD_SECRETS, names separated by spaces for one build, now name the secrets a build gets. The build reads only those, after every other check as before; the others are not read and never reach eb. An empty list names none. A name that is not a variable a secret may set, or that no secret in force has, is a configuration error that says where it was set. Without either, a build gets every secret, as before. What changes: - eb-local config shows a build.secrets row, "(every secret)" when nothing sets it. - eb-local config --env prints EB_LOCAL_BUILD_SECRETS when build.secrets is set. - eb-local --explain RECIPE says which secrets stay out of eb's environment when build.secrets leaves any out. - eb-local doctor says FAIL, with a site file, only for an unreadable secret that builds get; one they do not get is a warning, and a readable one is info. When nothing sets build.secrets, its FAIL line for an unreadable secret now says to name the secrets builds need. - The usage text says how a build's secrets are chosen. The README and a new how-to page describe it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A build and
eb-local ebcheck, once EESSI-extend has loaded, that the install is where the module put it under Lmod 8.6.19: under<prefix>/versionsfor a build, or in the tree the install variable selects foreb-local eb. They stop with exit 2, beforeebruns, whenEASYBUILD_INSTALLPATHis outside that tree, or whenEASYBUILD_INSTALLPATH_SOFTWAREorEASYBUILD_INSTALLPATH_MODULESis set. A leftoverEESSI_USER_INSTALLnext toEESSI_PROJECT_INSTALLis refused, because EESSI-extend then makes a private user install. Both warn when loading EESSI-extend sets or unsets an install variable, and a build names each install variable it clears from the caller's shell.The refusals of a node-local mount and of an install prefix, by a build,
eb-local cleanandeb-local doctor, say what was refused and why, what the command then did not do, and, on the next line, what to change. A workspace that cannot be made names the error the kernel gave, where it used to sayAccessDeniedfor every failure, and the umask is no longer left cleared when the path is too long.build.secrets, in a configuration file or a profile, andEB_LOCAL_BUILD_SECRETS, for one build, name the secrets a build gets. The build reads only those; a secret it does not get never reacheseb. Without either, every build gets every secret, as before.The tests pass on x86_64. The aarch64 emulation failures are those of the base.